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Abstract. In Camot groups of step< 3, all subriemannian geodesies are proved to be nor- 
mal. The proof is based on a reduction argument and the Goh condition for minimality of 
'^v^' ' singular curves. The Goh condition is deduced from a reformulation and a calculus of the 

end-point mapping which boils down to the graded structures of Camot groups. 

o 

^ . 

pL| . 1. Introduction 

■ This paper is inspired by the smoothness problem of subriemannian geodesies, one of 
the fundamental problems in subriemannian geometry. We first study the case of Camot 
group with step< 3. In this case we proved that all subriemannian geodesies are normal 
and thus smooth. 

To state the subriemannian geodesic problem, we first recall some basic facts on sub- 
riemannian geometry. We refer to the book [26] for detail. A subriemannian manifold 
^ ' is a smooth n-dimensional manifold M with a ^-dimensional subbundle or distribution 
A c TM on which a smooth inner product g,. is endowed. (A, g^) is called a subriemannian 
structure on M and A horizontal bundle. In this paper, we always assume M is connected 

■ and A satisfies the so-called Chow-Homander condition which means that vector fields of 
. A together with all their commutators span the tangent space at each point on M. Camot 

^ I groups are important examples of subriemannian manifolds. A Camot group G is a con- 

nected, simply connected Lie group with a graded Lie algebra 



o 



oo 

o 



(1.1) b = V^®---®V'', withV' = [V\V'-^],[V\V] = 0J = 2,-- - ,r. 

O ■ 

The integer r is called the step of G. Since G is connected and simply connected, the 
exponential map from t> to G gives a global chart for G. Camot groups are tangent spaces 
(in the sense of Gromov-Hausdorff) of equiregular subriemannian manifolds, see [1251 
^ . It is believed that the role played by Camot groups in subriemannian geometry is similar to 
i the role of Euclidean Spaces in Riemannian geometry. 

It follows from the Chow-Rashevskii connectivity theorem that for any given points 
p,q e M there always exists at least a horizontal curve connecting p and q, see [[T6ll29l . 
Here a horizontal curve is by definition an absolutely continuous curve y : [0, 1] — > M such 
that y{t) 6 Ay(,)M whenever y{t) exists. Thus one can define a natural distance: 



f 

Jo 



(p,q) = inf cir, y)dt 
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where the infimum is taken among the set Q.(p,q) of all horizontal curves y such that 
7(0) = p and y(l) = q. d^^ is called the Camot-Caratheodory distance of (M, A,^^). A 
subriemannian geodesic is a horizontal curve locally realizing d^^- We will reserve the 
terminology "minimizing geodesic" or "minimizer" for those globally distance-realizing 
subriemannian geodesies. It is not difficult to prove that any two sufficiently close points 
can be joined by a minimizing geodesic. If (M, d^,.) is complete, there is a minimizing geo- 
desic connecting any two points. Before Montgomery [|27l (in 1991) discovered a smooth 
singular minimizer, it was taken for grant (see e.g. (30]) that each subriemannaian geodesic 
similar to a Riemannian geodesic could satisfy a Hamilton- Jacobi equation: 

(1.2) ^ = ^, A. = JJi 

where (x', /I,) is a coordinate system of T*M, H(x, A) = maXveA, {/^(v)-^^f(v, v)} (A e T*M) 
is the subriemannian Hamiltonian. A horizontal curve y{t) = {x'{t)) (denoted by a local co- 
ordinate) satisfying (11.21 ) almost everywhere for some lift A{t) = (/l,(0) can be proved to be 
locally minimizing and smooth, and is called a normal geodesic. Montgomery's example 
shows that not all subriemannian geodesies are normal. The subriemannian geodesic prob- 
lem is a special case of geometric control problems. In fact singular curves or abnormal 
extremals play a very important role in optimal control theory, see e.g. [fT2l |51. It is well 
known that the Pontryagin Maximum Principle (or the Lagrange Multiplier Rule in the 
Lagrangian formulation) gives the first order necessary condition of optimality for optimal 
control problems. This first order condition is hardly considered to be satisfactory when 
one studies abnormal extremals. Recently experts developed necessary/sufficient second 
order conditions of optimality, i.e, Goh condition and generalized Legendre-Jacobi con- 
dition, see e.g. JH [51 [6l |71 These conditions were derived from the finiteness of the 
generalized Morse index of critical points of the end-point mapping. We refer to [|6l [H |51 
for the finiteness of the generalized Morse index. 

Let 0.{p) be the set of all horizontal curves y : [0, 1] ^ M such that 7(0) = p. It 
is well known that 0.{p) is a smooth Banach manifold, see [fTOll . The end-point map- 
ping is the smooth mapping S : Q.{p) M defined by taking y e 0.{p) to 7(1). Thus 
Q.{p,q) = <§~^{q). In general S' is not regular at all curves in 0.{p) and thus Q.{p,q) possi- 
bly has no smooth structure if ^ is a critical value of S'. If 7 e ^{p) is a critical point of 
S', we call 7 a singular curve. After Montgomery's example, Liu-Sussmann in [i23l gave 
more examples of singular curves which are locally minimizing. All these examples found 
on rank two distributions are in fact -rigid curves which by definition are locally isolated 
curves in Q.(p, q) with respect to the C' -topology, see also Bryant-Hsu [fT3||. For every rank 
two distribution satisfying A^i^A^atpeM they proved that there exists a rigid curve 
locally length-minimizing and emanating from p, see also Agrachev-Sarychev in flU Theo- 
rem 6.2]. Here = A, A' = A'"^ -I- [a\ A'"'] for / = 2, • • • . The research of such curves first 
appeared in the work of Caratheodory, Engel, and Hilbert, see [ lTl[32]| . Classical calculus 
of variations can not fully deal with the subriemannian geodesic problem when Q.{p, q) con- 
tains singular curves, because there possibly exist no smooth variations of such curves. But 
singular curves could be subriemannian geodesies as shown by the above mentioned work. 
A fundamental problem is whether all subriemannian geodesies are smooth. This prob- 
lem is equivalent to the question whether all singular geodesies are smooth, since normal 
(regular) geodesies are always smooth. There are some substantial results so far, while the 
problem is still open for general cases. Gole-Karidi in [fTSTl constructed the first example 
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of a Camot group with a strictly singular minimizer and they proved that all subrieman- 
nian geodesies in a step-two Camot group are normal. Agrachev-Sarychev [|71 proved that 
there admit no strictly singular geodesies for medium fat distributions including strong- 
generating distributions (fat distributions) for which Strichartz [|30l had already obtained 
the conclusion. For a class of equiregular subriemannian manifolds, Leonardi-Monti [|22l 
showed that length-minimizing curves have no comer-like singularities which in particular 
implies that all singular geodesies in Camot groups of rank two with step< 4 are smooth. 
There are also some "generic" results which claims that for 3 < k < n there exists an open 
dense subset Ok of the space consisting of all k-dimensional distributions on M (en- 
dowed with the Whitney C°° topology), such that each distribution in admits no singular 
geodesies, see flU [T51 and references therein. 

In this paper we will concentrate on the case of Camot groups. As mentioned above 
Camot groups are very important in subriemannian geometry. Our study here will be in- 
structive for later considerations of subriemannian geodesies of general distributions. Our 
starting point is the refined formulation of the end-point mapping which boils down to the 
graded structures of Camot groups. The graded structure (II .11 ) implies that each horizon- 
tal curve T = iy\y^, - ■ ■ ,7') is uniquely determined by the first layer y^. Here we use 
the exponential mapping exp to identify the Camot group G with its Lie algebra b and 
y' = :7r'(exp"' Y) e V' where n' : b = ® ■ ■ ■ ® V ^ V is the projection to the i-th 
component. The subriemannian geodesic problem in Camot groups can be formulated as a 
minimization problem with equality constraint. The end-point mapping ^ is different from 
the ordinary one which usually takes a control function to the end point of the trajectory 
uniquely determined by the control function (the initial point is fixed). The formula for 
the differential of S' can be written out for Camot groups with any step by a very tedious 
computation. In the case of step 3, the differential and the intrinsic Hessian of the end- 
point mapping are of simple form. We will get the second order necessary conditions for 
optimality of a singular curve, that is, if a singular curve Y = {y\y^, - ■ ■ , y') is locally 
energy-minimizing then y' must satisfy the Goh condition and the generalized Legendre- 
Jacobi condition, see Proposition 14.31 and 14.51 From the Goh condition and the graded 
structure (11.11 ) we deduce that the first layer of a singular geodesic Y must be in a lower- 
dimension subspace. Thus singular geodesies must be in Camot subgroups of rank 2 or 
step 2. We finally reduce the problem to the rank two case which (known to experts) is 
easy, see Theorem 15.21 and Theorem 15.41 The reduction to lower subgroups is a curious 
coincidence with Hamenstadt's suggestion for the smoothness problem, see [|T9l . 

The paper is organized into five sections. In the next section we give a formulation of 
the end-point mapping which is based on a characterization of horizontal curves in Camot 
groups. Section 3 is devoted to the calculus of the end-point mapping. We will give the 
differential, the intrinsic Hessian for the step< 3 case. In Section 4 we derive the second 
order necessary conditions of singular geodesies. We prove the main results in Section 5. 

2. Horizontal Curves and the End- Point Mapping 

2.1. Basic Structure of Carnot groups. Let G be a Camot group with a Lie algebra b sat- 
isfying (11.11) (we call such Lie algebras Camot algebras). We endow on an inner product 
<■,•>. Let Hi = dim(y,), n = 2;=i ni. The r-vector (ni,ni +n2, • • • , • • • , n) is called 

the growth vector of the Camot group. We fix an orthonormal basis of V^, {ei, ■ • ■ , e„i}, 
then arbitrarily extend it to a basis of b, {ei, ■ • ■ , e„i, e„|+i ■ • ■ , e,,}, and extend < •, • > to 
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an inner product ^ on b making the basis orthonormal. Via the exponential mapping exp 
we identify G with b or R" with a group law determined by the Baker-Campbell-HausdorfF 
formula. For p e G, setting Xi(p) = ^ {p ■ exp(?e,)}, i = 1 • • • , n, we get the basis of the 
space of left-invariant vector fields. The horizontal bundle A = span{X\, ■ ■ ■ ,X„^} satisfies 
the Chow-Hormander condition by (11.11 ). Let gdiXiip), Xj{p)) =< e,, ej >,i,j = I,- ■ ■ ,ni. 
Thus we have a subriemannian structure (A,^^) on G. We also extend to a left-invariant 
Riemannian metric gr such that {Xi, ■ ■ ■ ,X„} is an orthonormal basis of TG. We emphasize 
that subriemannian geodesies in G are independent of the choice of orthonormal bases and 
their extensions, that is, they are completely determined by (b, y^ < ■, • >) or equivalently 
by (G,A,^,). 

Example 2.1. 1 . The simplest Camot group is the Heisenberg group H'" with the Heisen- 
berg algebra (growth vector=(2m, 2m -I- 1)) Ih = span{ei, ■ ■ ■ , e„,, /i • • • , /„,} © span{^} with 
the basis satisfying that [e,, fi] = g, i = 1, • • • , m, and all other brackets vanish. 

2. The Engel group is a Camot group with the growth vector (2,3,4). Its algebra is 
span{ei, e2}®span{e3}©span{e4} with [61,62] = 63,161,63] = 64. Note that Camot groups of 
rank two (ni = 2) has a special feature: the second layer has dimension 1 whatever its step. 
We will see later this feature make the study of its subriemannian geodesies very easy. 

3. The free Camot group with bi-dimension (k, r) has the maximal vector growth among 
all Camot groups with k generators and step=r. The free Camot groups play some particu- 
lar roles in nilpotent analysis. 

4. Let b be a Camot algebra satisfying (11.11 ). If c VMs a lower-dimensional subspace, 
then b = © ■ ■ ■ © W'Ms a Lie subalgebra of b, where = W, W' = [W\ W'-^], i = 
2, • • • , r, and r G [1, r] is the largest integer such that W 0. It is obvious that b is a 
Camot algebra and G := exp(b) is a Camot subgroup of G (we regard Euclidean spaces 
as abelian Camot groups). We use b(W) (resp. G{W)) to indicate the Camot subalgebra 
(resp. Camot subgroup) generated by W. The reduction to Camot subgroups with lower- 
dimensional first layer is one of main tricks in this paper. 

Recall that the differential of the exponential mapping is given by the following formula 

(2.1) d exp(e) = Id - y ^— ^ad(e)'"-i , 

ml 

see e.g. [31J. For an absolutely continuous curve T : [0, 1] — > G, we denote by 7 the 
corresponding curve exp"'(T) with values in the Lie algebra b. We have 7 = £[=i7' where 
y = n'(y), n' : b ^ V is the projection to the i-th layer. It is obvious that 7 is also 
absolutely continuous. From (12.11 ) we get for a.e. t e [0, 1] 



(2.2) 



m = m - y ^^ad(7(o)'"-'(r(0) 

^ m\ 

m=2 
" (-1)" 



— r[r(0,r(0]m-i 

m! 

m=2 



In the last formula we used the iterated Lie bracket which is defined by 
[e, f]m = [e, [6,[--- ,[6, /],],■■■ ,] and [e, /]o = /. 
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From (|2.2I) we obtain that Y is horizontal if and only if for a.e. t £ [0, 1] 



n 



7(0 - 2 



m=2 



(-1)" 

ml 



[7(0,7(0] 



m-l 



0, i = 2, 



r. 



We summarize as 

Lemma 2.2. An absolutely curve T in G is horizontal if and only if for a.e. t e [0, 1] 

(-1)" 



7 



m=2 



m: 



-^'([7(0, 7(0]m-i), / = 2, • • • , r. 



We denote by TY^ the Sobolev type space of all horizontal curves T : [0, 1] — > G with 
square integrable derivatives. In the rest of the paper we assume all horizontal curves in G 
are in "H^ . For our purpose this assumption is not restrictive since all rectifiable curves can 
be arc-length parameterized. Combining (|2.21) with Lemma IZ2l we have for a.e. t e [0, 1] 



(2.3) 



ta) = J]im(T(o) 



i=l 



where 7^(0 = '^'!l^x'(t)ei. Define P : G ^ V\ Pip) = n\&x^-\p)). The mapping V is 
just the projection R" 3 (jci, • • • , jc„,, • • • ,x„) — > {x\, - • • e R"' when we identify G 
as (R", •)■ The formula (12.31 ) in particular implies that !P is a Riemannian submersion from 
(G, gr) to < ■, • >) (or equivalently from (R", gr) to (R"' , < •, ■ >)) with the property that 
for any peG, Pi,,p{Xi{p)) = e,-, z = 1, • • • , nj, and P^^piXj(p)) = for j = n^ + I, • • • ,n. 

Note that the graded condition (11.11 ) for the Lie algebra b is equivalent to the following 
condition 



(2.4) 



y = [V\ [■■■ , [V\ V^]]], i = 2, ...r, and = for j > r, 



which together with Lemma |Z2] implies that 
6(-l)' ' 



(2.5) y = J] 



m=2 



m\ 



for z = 2, • • • , r, a.e. 



which means that 7^ determines 7^, 7^, • • ■ ,y'' recursively. We list 7^, 7^, 7"^ as functions of 



r 



(2.6) 



'7' = ^[7^7'] 



y^ = \ {[y ,7'] + [7',7'] + [7',^]) - \ [Vy\ [7^7']] + [7^ [7',7']] 
+ Vy\Vy\y% + ^Vy\y%. 



Sometimes we will abuse the notation T = (7^ • ■ ■ , 7^ or T = £-=i7'. 
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Proposition 2.3. (1) Given p e G. Every absolutely continuous curve y' : [0, 1] — > has 
a unique horizontal lift Y = (y^ • • • , y') : [0, 1] ^ G determined by (\2.5\l with T(0) = p. 
They have the same length and same regularity or smoothness; (2) Horizontal lifts of every 
straight line (or its interval) in are subriemannian minimizing geodesies. 

Proof. (1) Note that the class of absolutely continuous curves in b is just the Sobolev 
class W^^'^([0, 1], b). This implies y' is continuous and thus bounded in [0, 1] with y^ e 
L\[0, 1], b), see e.g. [14, Chapter 2]. So there exists y^ e H^'-'([0, 1], b) such that y^(t) = 
^ X 7^]'^'^ + ^^(exp"^ p). Continuing this process, and noting that each summand in the 
right hand side of (12.51 ) contains only one term with derivative and other terms are bounded, 
that is, the right hand side of (12.51) is in L'([0, 1], b). Thus the function y' satisfying 



dr + ;r'(exp ' p) 



is in V7^''([0, 1], b). From (|2.3I) y' and its lift above have the same length. 

To see (2), we recall by definition that in (G,^^) Riemannian geodesies which are hor- 
izontal must be subriemannian geodesies. Since the geodesies of Euclidean space (V^ < 
•, ■ >) are straight lines (or their intervals) , their horizontal lifts are Riemannian geodesies 
because f is a Riemannian submersion, see [28]. The minimizing property is obvious. □ 

By Proposition 123] we sometimes do not distinguish a horizontal curve T = (y^ • • • , y'^) 
with its projection to the first layer y ' . 

Note that horizontal lifts of straight lines in are not necessarily still a line (looking in 
b) if G with step> 3. In fact, let j^(t) = + Vq with v, Vq e By the formula (|2.6I) we 
have 

' 9 1 
7 (0 = i^v,vq\ 

y\t) = ^[v, [V, Vo]]? - ^[vo, [vo, V]] + \\y\Q), v]. 

So the third layer of the lift is not a line unless [v, [v, Vq]] = 0. While if vq = 0, that is, the 
line passes through the origin, its lift is just itself. 

2.2. The End- Point Mapping. Given p,q e G, deonte by ^{p) the Hilbert manifold of 
all horizontal curves T e^H^ with Y(0) = p. Let Q.{p,q) = {Y e Q.{p) : Y(l) = q). Since 
Q(p) = p ■ Q(0) := {p ■ Y : Y G i^(0)}, Q.{p, q) = p ■ QiO, p'^ ■ q) and the metric g, is left- 
invariant, it suffices to consider horizontal curves emanating from the unit. Here we abuse 
to denote the unit of G. From Proposition l2.3[ we see that the projection V gives a bijective 
mapping (still denoted by V) from Q(0) to H\0) := {y^ e H\{0, 1], V^) : y\0) = 0} with 
the mapping of horizontal lift as its inverse, where H^([0, 1], V^) denote the Sobolev space 
of all absolutely curves y' : [0, 1] — > with square integrable derivatives. Note from the 
formula (O, we have for Y e Q(0), t e [0, 1], 

m = (r\t),y\t),--- ,Y{t)) 

where y'it), i = 2,- ■ ■ ,r is regard as a mapping F''^ (defined recursively) from H^(0) to V: 



Jet ' / 1 \m I 
^—0 ^* I u t" — r" 



dr. 



m=2 \jl+jl+-+jm=i 
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Now the original end-point mapping 

(2.8) end : ^^(0) 9 T ^ Y(l) e G 
can be interpreted as 

(2.9) <g:H\Q)3y^ {F\y\ F\y'\ ■■■ , F^y')) 6 b 

where F\y^) = y^{\), F' = F''\i = 2,- ■ ■ ,r. 

Noting that given exp^ = ^ e G for ^ e b, ^^(0, ^) = exp(i^'"H^)), the subriemannian 
geodesic problem in G 

1 ^1 



min - gc(r,r)dt 

Ten(0,9) 2 Jo 

is equivalent to the minimizing problem with equality constraint 

1 

(2.10) min - \yfdt. 

^(r')=f 2 Jo ' 

By the Cauchy-Schwarz inequality the problem of minimizing the energy functional is 
equivalent to that of minimizing the length functional. The existence of the subriemannian 
geodesic problem even for general subriemannian manifolds can be obtained by an argu- 
ment of direct method in calculus of variations, see e.g. Appendix D in [26J where one also 
will find the ordinary formulation of the end-point mapping. What we are concerned with 
is their smoothness. The refined mapping : //'(O) — > b from a Hilbert space to a vector 
space will help us much. 

3. The Calculus Of The End-Point Mapping 

3.1. The Generalized Morse Index Theorem. To begin some computation, let us first 
see what we need according to the theory of generalized Morse index which we will later 
resort to. Let {X, \\ ■ ||) be a Banach space, L : X ^ R, and F : X ^ Y with Y a finite 
dimensional vector space, be Frechet differentiable mappings. Given yo e Y, consider 
the minimizing problem with equality constraint 

(3.1) min L{x). 

The Lagrange Multiplier Rule states that if x 6 X is a solution of (13.11 ) then there exists a 
nontrivial couple (/l°, /I*) e R x Y* such that A'^dj^L + A*dj,F = 0, where d.^L (resp. d^F) 
denotes the Frechet derivative of L (resp. F) at the point x. In other words, the point x is a 
singular point of the augmented end-point mapping £, : X 3 y ^ iL(y), F{y)) 6 R. x 7 and 

(3.2) A ■ d,£ = A^djcL + A*d,,F = with A = (/, A*). 

The abnormal case A'^ = arises exactly when Imid^F) ^ Y, i.e., jc is a singular point of F. 
In this case we call {x. A*) an abnormal extremal. In the regular case lm{djfF) = Y, we take 
/l° = 1. For X e X if there exists A* such that 

(3.3) d,L + A*d,F = 0, 

we call (x. A*) a normal extremal. The definition of normal extremals is equivalent to the 
one given in the Introduction. In the theory of subriemannian geodesies there exists a 
correspondence between A* in (|3.3I) and the Hamiltonian lift A{t) in (11.21) . see [|20ll or [|26l 
Chapter 5]. We remark that an abnormal extremal may be normal by choosing a suitable 
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multiplier (or a Hamiltonian lift). Those abnormal extremals which can not be normal for 
any multiplier are called strictly abnormal extremals. 

The corank of x is defined as the codimension of lm(d^£.). The following theorem, 
which gives necessary/sufficient conditions of optimality for a singular point is enough for 
our purpose, for the general versions see [5. Chapter 20]. 

Theorem 3.1 ([H|7l|6l|5l). If x is a local minimizer in X of the minimizing problem di.iP . 
of corank N, then for the nontrivial pair of Lagrange multiplier A = (/l", A*) (A^ = or I) 
satisfying A3.2\l . the Morse index of the quadratic form A ■ d^X restricted to kerdj^F is less 
than or equal to N - 1. 

We recall that the Morse index of a quadratic form is the maximal dimension of sub- 
spaces on which the quadratic form is negative definite. Theorem 13. II is classical for the 
regular case for which (|3.3t is the Euler-Lagrange equation, see |[24l . 

3.2. The Differential of The End-Point Mapping. In the next section we will derive 
second order necessary conditions for minimality of abnormal extremals of the problem 
(12.101 ) from the finiteness of the Morse index of the quadratic form l-d^X stated in Theorem 
13. 1[ In the following we do some computation of the differential of the end-point mapping. 

Lemma 3.2. Given T = (y\ •••,/) 6 ^^(0), then Ty^iO) = Ty^H\0) = H\0). 

Proof H^{0) is a Hilbert space. Let 6 H^{0). The family of horizontal lifts of + ecp 

(e G [-60, 6q\), 

T,(0 = {y\t) + 60(0, F^\y' + 60), • • • , F'-\y' + e 



where F'-*, i = 2 - ■ ■ , r, is defined as in (12.71) . is a smooth curve in 0.(0) with To = T. 

On the other hand, if is a smooth family in ^^(0) with To = T, then by Lemma 12.21 
and (1231) . (lO) we have 

r,(t) = {yl,F'^\yl),--- ,F^''{yl)) 

where y] = is a smooth family in H\Q) with y\^=y^ = f'(T). □ 

For y\(p e H^(0), the difi'erential at y^ of the end-point mapping is 

(3.4) dy^ : H\0) 3(P^ dy A0) = ((pa),d^^F\(f>), ■ ■ ■ ,dyF''(0)) e T,G 

where q = {y^{\),F^{y^), ■ ■ ■ ,F''{y^)) and F' is shortened for / = 2, • • • ,r. Noting 
that the differential dyiF'(0), / = 2, • • • , r, recursively depends on dyiF^''((f>), d^iF^''(0), j = 
2, 1, ? e (0, 1], its computation is complicated for i > 5. We restrict to the case of 

step< 3 partly also because of technical difficulties in the next two sections. Observe that 
^d^iF '(0) = d^iF'''(0). From ^M-^J} we have for step=2 



(3.5) 



F''{y') = ^J^^[y\r]dT 
dy.F^^cP) = ^ [(P,y']dT + ^[y\t),(f>(t)] 

dy.F^\cp) = ^-mxy'm + ^[y(t),m 
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For step=3, from (|2.6I ) we first have 

F^' = y\t) = \^\F^\y\t)] + Uy\t), {y\t),y\t)]\, 
2 at 3 

then using (13.51) get 



(3.6) 



Jo ^ LJo 

+ ^[^(0, [7^0,0(0]] 



1 













[7^r^]JT,0(^) 



In the above computation we used integration by parts, skew-symmetry and Jacobi identity 
of Lie brackets to arrange terms. 

Lemma 3.3. In the case r = 3, by {\3.4\) - ^JM we have (p e ker (d^i^fj if and only if 

0(1) = 01 



(3.7) 



Jo 



[(p,y^]dt = 



Jo 



Now we compute the second Frechet derivative of the end-point mapping <f for r = 3. 
From (|3.5I) we have 

(3.8) d^,F2(0,0) = ^ 
For (f) e ker (d^i t?) it follows from K6\i and (13771) that 

(3.9) dl,F\(f,,c/>)= [ [4,,[(f>,y']]dt+ [ 

Jo Jo 



y-^y(i),[0,0] 



dt. 



So the intrinsic quadratic mapping (see e.g. BH p. 294-296]) of ^ for r = 3 

d^,(f : ker(dy (?) x ker(dy (?) ^ b 

is 



(3.10) d^,A0,0) = |O 



, r [0,0] jf, r 

Jo Jo 



1 

,•,1 



Jo 



1 



[0, [(p,r]]dt+ 

4. GoH Condition and Legendre- Jacobi Condition 



y--y(i),[0,0] 



dt 



In the rest of the paper we assume the Camot group has step< 3. From (|3.5I ) and (|3.6I) 
we have for e //^O) with 0(1) = 

(4.1) dyA0) = (o,^[0,y]j?, j| y-^y(i),[0,y] 

Applying the Lagrange Multiplier Rule to the problem (12.101 ) we get 
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Proposition 4.1. If e H^{0) is a minimizer of the problem d2.iOD . then there exists 
1 = iA\A^, A^) e b* with A' e (V')*, i= 1,2,3, such that for any cp e H\0) 



r-l r 1 



(4 



or 



(4.3) 



dt = Q with 0(1) = 



ld^i^(0) = iV(l) + A^Ay^F^-{(l)) + iM^iF^(0) = with 1^0. 



Proof. Taking L as the energy functional L{y^) = j J^^ ly^l^dt, by the Lagrange Multiplier 
Rule there exists nontrivial (A^, A) eRxb* with /l" = or 1 and A = (A\A^, A^) such that 
for any e H(0) 

A\iL{(()) + /lV(l) + A%iF^{(()) + A^dyiF\(P) = 0. 

When A^ = I, (|431) follows from the last formula, (|4TI ) and dyiL(0) = j^' < 7^ (/> > J?. □ 

We call (or its horizontal lift) satisfying (|4.21 ) for some (/l^, /l^) a normal geodesic. By 
a standard argument from the theory of (elliptic) differential equations normal geodesies 
are smooth, see e.g. [14]. Singular geodesies are local minimizers y^ (or their horizontal 
lifts) of the problem (12.101) satisfying (14.31) for some A = {A\A^,A^) G b*. For singular 
geodesies the following result follows from Theorem 13. H and (13.101 ). 

Proposition 4.2. Ify^ e H^(0) is a singular minimizer of the problem A2.10\) . there exists a 
nontrivial A = {A\A^, A^) satisfying ( 14.31) such that the Morse index of the quadratic form 



~,a2 r x\(p,4>]dt+ r A' 

Jo Jo 

+ f A\(p,[(p,y^]]dt (0 6ker(dy(f)) 
-Jo 



(4.4) 



is finite. 



y'-^y\l),[cp,4>] 



dt 



Proposition 4.3. Let y^,A = (A^,A^, A^) be as in Proposition \4.2\ Assume y^ is parameter- 
ized proportionally to arc-length. Then 



(4.5) 







A^ [y\t), [a, b]] = 0, Va, b e V\yt e [0, 1]. 



Proof. The argument is similar to [HI Proposition 20.13]. The idea is a type of scaling or 
blowing up method. 

Let t G [0, 1] be a Lebesgue point of y^. Take a smooth mapping c : R. — > with 
support on [0, 2n] such that c(s)ds = 0. Let (pc^e{T) = c(J-j-)df with e small. This 
certainly implies (pcei^) = c(-^) for r G [0, 1] and 0c,e £ H\0) with 0c,e(l) = 0. Letting 
w(s) = £ c(s)ds, we have 



(4.6) 



I 



1 p2n 



I 

Jo 



A [w{s), c{s)] ds 
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and similarly 



(4.7) 



Jo 



■In 

2 / o3 



Jo 



dt=e' I A 

+ 6^0(1) 



y\f)-\y\mw{s),c{s)] 



ds 



where we used the fact y 6 L°° and thus 7^(r + es) = y^{T) + 60(1) for e small enough. For 
the last term in (14.41 ) we have 

[0.,o y'm = 6^ [w{s\ [w{s\ y\f + 65)]] ds = 6^0(1). 

From (|431)-(I48]). we get 

co{w{s),c{s))ds + e^O{\) 



(4.9) 



where a»(a, b) = A^[a, b] + A^ ^y^ir) - ^y\l), [a, b]^ is a skew-symmetric bilinear form on 

We claim that to = 0. In fact, if a> 0, then ranka* = 2Zo > and we can change the 
basis of such that 



!=1 



for any Z? = (x\ • • • , x"^), a = (y\ • • • , j"') 6 V^. Now we take 

c(s) = (x\s),0, - ■ ■ ,0,x''>^\s),0, - ■ ■ ,0) 

where x\s) = Zti^k cos ks,x''^\s) = ZZinkSinks, and (a)~ (t/^)^^! e iK Putting 
c(s), w(s) = c(s)ds into (14.91 ) we get 



/ CO 



ld^l(f(0c,e,0c,e) = - 



6^ + 6^0(1). 



\ k=i ) 

From the last formula and the construction of 4>c,e it follows that there exists an infinite 
dimensional space 7C such that ld^,(f (0, 0) < for each e 7C. Note that n ker^d^iff ) 
is also infinite dimensional, since the rank of S is less than n. It implies the Morse index 
of Ad^iS' is infinite. This is impossible by Proposition |4]2l so a» = 0. 
We have proved that if ? 6 [0, 1] is a Lebesgue point of y\ then 



(4.10) 



A\a, b] + A^ 



y\t)-^y\l\ [a,b] 



= for any a,b eV^. 



Since almost all points in [0, 1] are Lebesgue points of y^ e U° by the Lebesgue differ- 
entiation theorem, it follows from the continuity of y^ that (|4.10l) holds for any t e [0, 1]. 

" = (since y\Q) = 0). Comb- 

= for any t e [0, 1] and any 



In (HTTOl) letting ? = we get A\a,b] + A^ [-\y\l), [a,b] 
ing the last identity with (|4.10l) . we obtain A^ \y^it), [a, b] 
a,b e V^. Applying the identity (|4.10l) again, we finally have A^^ = for any ^ 6 V^, since 



□ 
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The condition of (14.51) is called the Goh condition which first appeared in references of 
singular control theory, see [fTTI . A curve 6 H^(0) (or its horizontal lift) together with 
some nonzero A = {A\ A^, A^) e b* satisfying the Goh condition is called a Goh curve and 
the pair {jKA) is called a Goh extremal. Note that for a Goh extremal {y\A), A^ and 
(14.31) automatically holds by choosing A^ = 0. The following fact is instructive for the study 
of subriemannian geodesies even for general case. 

Corollary 4.4. Assume G is a Carnot group of step 3, with a Carnot algebra b = V ' © © 
V^. Let W be a lower-dimensional subspace ofV^. 

(1) If[W, c V\ then any curve (H^O) 3)y^ c W is a Goh curve. 

(2) Ifb is a free Carnot algebra, then any curve {H^(0) 3)y^ c W is a Goh curve. 

Proof. (1) By assumption dim(y^\[V7, V^]) > 1. For any curve {H^iO) 3)y^ c W choosing 
A^ = 0,A^ = and A^ annihilating [W, V^], we conclude that {y^ ,~A) is a Goh extremal. 
The statement of (2) follows from (1) and the fact that for a free Carnot algebra, [W, V^] c 
always holds when dim W < dim . □ 

Corollary I4.4l implies that each Carnot group of step 3 admits Goh curves. The following 
necessary condition is not used in this paper, but we include it for completeness. 

Proposition 4.5. Let y\ ~A = {A^, A^, A^) be as in Proposition \4.3\ Then 

(4.11) A^ [a, [a,y(0]] > Ofora.e. t e [0, 1] and any a e 
and 

(4.12) ld;,A0,0)= r A'[(/>,[(p,V]]dt>0, 0eker(d^.^), 



changing A to -A if necessary. 

Proof. It suffices to prove that (|4.1 11 ) holds for all Lebesgue points of y^ . 

Let f e [0, 1] be a Lebesgue point of y\ Assume that A^ [a, [a,7^(t)j < for some 
a eVK We choose a suitable basis of to diagonalize the quadratic form 



"1 



A'[a,[a,y\f)] = Yj'^i(x^)\ a = (x\- ■ ■ ,x"') 
with at least one term cr, < 0. For any smooth x : R — > R"' with support in [0,1], let 

Mt) = 

then we have 



0^-^,40,0,- ■• ,0 

1, (i'—l) terais 



(4.13) Ad\c^((l>„(f>,) = (ri [ x\t)dt<0. 

^ Jo 

Denote by 11 the set of all smooth mappings x : [0, 1] — > R"' with support in [0,1] and 
satisfying (14.131) . 11 is infinitely dimensional, so is ^(p^ : (p^ e ker^d^itfj ,x e flj. It is a 
contraction by Proposition 14.21 □ 

(|4.1 II) and (14.121) are called Generalized Legendre-Jacobi condition. 
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5. SUBRIEMANNIAN GeODESICS OF STEP 3 

In this section we assume G is of step= 3. 

Lemma 5.1. Any line (or its interval) through is a normal geodesic. 

Proof. For y^{t) = Ctvo, where C a constant and vq e y\ we take = 0,A^ = 0, then (14.21) 
holds for any (p e H\0) with 0(1) = 0. □ 

The following result on rank 2 case is well known, see e.g. [H]. For completeness we 
give a self-contained proof. 

Theorem 5.2 (Rank 2 Case). Let G be of rank 2, i.e., n\ = 2. Assume = span{ei, 62}- 
(\)In the case of the Engel group whose algebra b = spanfei, e2}ffispan{e3}®span{e4} with 

[^1,^2] = ^3, [^1,^3] = ^4> there is a unique arc-length parameterized singular geodesic 

which is normal and tangent to e^, that is, (t) = te^. 

(2) To the other case where b = span{ei,e2} ® span{e3} ® span{e4,e5} with [^1,62] = 

^3, [61,63] = 64, [62,63] = 65, singular geodesies are exactly those lines {or their intervals) 

in through the origin. 

Proof. Let be a singular geodesic parameterized proportionally to arc -length and satis- 
fying (1431) with some \ = {X^,??, ??). From (1431) = 0. This together with (|43T ) implies 
that ^ 0. 

We claim that must be in a line. In fact, if this is not true, there must exist ?2 e 
(0, 1] such that = span{y'(?i),7H?2)} which together with (1431 ) implies that = 
because [y\ V^] = . A contradiction! So y'(0 = {c^e\ -I- (P'e2)t for constants c\ with 
(c^)^ -I- (c^)^ 4^ 0. In Proposition |23] we see they are shortest subriemannian geodesies. 

(1) If b is the Engel algebra, from ?? ^ and Ac'Xc'ext -I- c^eit,e^^ = 0, V? e [0, 1] we 
get c^ = 0, since [61, 63] = 64 and [62, 63] = 0. From (13.51 ) and (13.61 ) by direct computation 
we verify that y'(0 = c^62? (c^ 0) is a singular curve (choosing e.g. ~X = (0,0, 1)). By 
Lemma [STTl it is normal. 

(2) When b is the free case, by direct computation we have Im^d^i^fj b. In fact, by 

(13.61) for any e H^ifS) there exists a constant 6 such that d^iF^(0) = (5(c'64 -I- c^65). So all 
y^O = {c'ex -I- c^62)? are singular geodesies. By Lemma ISTl they are also normal. □ 

Lemma 5.3. Let y' (or its horizontal lift) be a subriemannian geodesic in G and be con- 
tained in a lower-dimensional subspace W c Y^. Ify^ is a normal geodesic in the Carnot 
subgroup G(V7) of step 2 or 3, then y^ is also normal in G. 

Proof. Assume G{W) has step 3. Because y^ c is normal in G{W), by (14.21 ) there exist 
p e [W, W]* and v 6 [W, [W, W]]* such that 



Jo 



<y\(f)> +p[(p,y] +v 



y-iy(i),[0,y] 



dt = 



T 

holds for any (p e H\[Q, 1], W) with 0(0) = 0(1) = 0. Let Y^ (resp. Y^) be orthogonally 
decomposed as {W, W] ® (resp. {W, {W, W]\ ® U^). Now we take = p e (v^)*, = 

V e (y^^ , that is, we extend p (resp. v) to Y^ (resp. Y^) by annihilating (resp. U^). It 

is obvious that {A'^,A^) satisfies (gj) for any G //'([0, 1], V') with 0(0) = 0(1) = 0. The 
case when G{W) has step 2 is similar. □ 
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Theorem 5.4 (General Case). All subriemannian minimizers in G are normal. 

Proof. (1) Let be a singular geodesic which is parameterized proportionally to arc-length 
and satisfies (14.31) for some A = {A\A^, A^). 

From (143]) and (|43]) we have A^ because [V\ V^] = V^, [V\ V^] = V\ We claim that 
7^ is contained in a lower-dimensional subspace WofV^. Otherwise, there are • • • , e 
(0, 1] such that = span{yH?i), ■ ■ ■ ,7\t„J] which together with (1431 ) implies A^ = 0. 

Thus 7i (or its horizontal lift) is a subriemannian geodesic in the Camot subgroup G{W) 
whose algebra isb = W®[W,W]® [W, [W, W]] or W ® [W, W] or W. 

(2) If b = W, this implies that the horizontal lift of is itself. By Proposition 12.31 the 
line through and y^{to) for any to e (0, 1] is the shortest subriemannian geodesic. So y^ 
must be an interval of a line through 0. 

(3) If b = © [W, W], then y^ is a subriemannian geodesic in a Camot group of step 2. 
So y^ is normal in G(V7). From Lemma l53l y^ is also normal in G. 

(4) If b = V7 ® [W, W] ® [V7, [V7, V7]], then y^ is also a subriemannian minimizer in G(W). 
If y' is regular in G(VF), then by Lemma [53] is also normal in G. If y^ is a singular 
geodesic in G(W) and dim > 3, we repeat the procedure from step (1), with G (resp. b) 
replaced by G(W) (resp. b). 

By finite steps we arrive at the case of rank 2. Our statement follows from Theorem 
O □ 

Remark 5.5. The smoothness of subriemannian geodesies is very close to the regularity of 
the subriemannian distance. In fact, the pointwise smoothness of the subriemannian dis- 
tance depends on the strict normalness and uniqueness of subriemannian geodesies. The 
subanalyticity of the subriemannian distance (or sphere) was usually derived from the ex- 
clusivity of Goh curves. We refer the readers to [GT] @] and references therein for this 
topic. In our case of step 3, as pointed out in Corollary 14.41 there typically exist Goh curves 
which are smooth even normal if they are shortest. Theorem 10 in [4J proved that the 
subriemannian distances of free Camot groups of step 3 are not subanalytic. 
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